EchoPro Python Workflow#

Import libraries and configure the Jupyter notebook#

import matplotlib.pyplot as plt

# Python EchoPro
import EchoPro

# Allows us to grab the SemiVariogram class so we can use its models
from EchoPro.computation import SemiVariogram as SV

# obtain all visualization routines
from EchoPro.visualization import plot_layered_points, plot_kriging_results

# Allows us to easily use matplotlib widgets in our Notebook
%matplotlib widget

EchoPro.__version__

'0.3.1'

Set up EchoPro for a specific survey year#

Initialize EchoPro object using configuration files#

initialization_config.yml – parameters independent of survey year
survey_year_2019_config.yml – parameters specific to survey year
source – Define the region of data to use e.g. US, CAN, US & CAN
exclude_age1 – States whether age 1 hake should be included in analysis.

%%time
survey_2019 = EchoPro.Survey(init_file_path='../config_files/initialization_config.yml',
                             survey_year_file_path='../config_files/survey_year_2019_config.yml',
                             source=3, 
                             exclude_age1=True)

A full check of the initialization file contents needs to be done!
A check of the survey year file contents needs to be done!
CPU times: user 7.8 ms, sys: 0 ns, total: 7.8 ms
Wall time: 7.97 ms

Load and process input data#

This data is stored in survey_2019

%%time 
survey_2019.load_survey_data() #file_type='biological')

CPU times: user 1.32 s, sys: 11.6 ms, total: 1.33 s
Wall time: 1.37 s

survey_2019.specimen_df.head()

	sex	length	weight	age
haul_num
1	1	24.0	0.08	1.0
1	1	23.0	0.06	1.0
1	1	22.0	0.06	1.0
1	1	21.0	0.06	1.0
1	2	22.0	0.06	1.0

Compute transect biomass and abundance, and corresponding densities#

Biomass, biomass density, abundance and numerical density results are stored in survey_2019.bio_calc.transect_results_gdf, survey_2019.bio_calc.transect_results_male_gdf and survey_2019.bio_calc.transect_results_female_gdf

%%time
survey_2019.compute_transect_results()

CPU times: user 1.76 s, sys: 7.03 ms, total: 1.77 s
Wall time: 1.77 s

survey_2019.bio_calc.transect_results_gdf.head()

	latitude	longitude	stratum_num	transect_spacing	geometry	numerical_density	numerical_density_adult	biomass_density	biomass_density_adult	interval	...	biomass_age_bin_14	biomass_age_bin_15	biomass_age_bin_16	biomass_age_bin_17	biomass_age_bin_18	biomass_age_bin_19	biomass_age_bin_20	biomass_age_bin_21	biomass_age_bin_22	NASC_adult
transect_num
1	34.397267	-121.143005	1	10.0	POINT (-121.14301 34.39727)	0.0	0.0	0.0	0.0	0.484267	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	34.397391	-121.133196	1	10.0	POINT (-121.13320 34.39739)	0.0	0.0	0.0	0.0	0.503849	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	34.397435	-121.123057	1	10.0	POINT (-121.12306 34.39744)	0.0	0.0	0.0	0.0	0.504074	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	34.397394	-121.112871	1	10.0	POINT (-121.11287 34.39739)	0.0	0.0	0.0	0.0	0.495015	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	34.397437	-121.102888	1	10.0	POINT (-121.10289 34.39744)	0.0	0.0	0.0	0.0	0.501398	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

5 rows × 38 columns

print("Adult Biomass Estimates without Kriging:")
print(f"Total : {1e-6*survey_2019.bio_calc.transect_results_gdf.biomass_adult.sum():.3f} kmt")
print(f"Total Male: {1e-6*survey_2019.bio_calc.transect_results_male_gdf.biomass_adult.sum():.3f} kmt")
print(f"Total Female: {1e-6*survey_2019.bio_calc.transect_results_female_gdf.biomass_adult.sum():.3f} kmt")

Adult Biomass Estimates without Kriging:
Total : 1643.221 kmt
Total Male: 815.167 kmt
Total Female: 827.732 kmt

Compute abundance and biomass for each length and age bin, over transects#

survey_2019.compute_length_age_variables(data="transect")

survey_2019.bio_calc.transect_bin_abundance_female_df.head()

	age_bin_1	age_bin_2	age_bin_3	age_bin_4	age_bin_5	age_bin_6	age_bin_7	age_bin_8	age_bin_9	age_bin_10	...	age_bin_14	age_bin_15	age_bin_16	age_bin_17	age_bin_18	age_bin_19	age_bin_20	age_bin_21	age_bin_22	un-aged
len_bin_2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_6	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_8	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_10	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

5 rows × 23 columns

survey_2019.bio_calc.transect_bin_biomass_female_df.head()

	age_bin_1	age_bin_2	age_bin_3	age_bin_4	age_bin_5	age_bin_6	age_bin_7	age_bin_8	age_bin_9	age_bin_10	...	age_bin_13	age_bin_14	age_bin_15	age_bin_16	age_bin_17	age_bin_18	age_bin_19	age_bin_20	age_bin_21	age_bin_22
len_bin_2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_6	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_8	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_10	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

5 rows × 22 columns

Jolly-Hampton CV Analysis#

Compute the mean of the Jolly-Hampton CV value on data that has not been Kriged
Note: the algorithm used to compute this value is random in nature

%%time
CV_JH_mean = survey_2019.run_cv_analysis(kriged_data=False)

CPU times: user 1.34 s, sys: 4.06 ms, total: 1.35 s
Wall time: 1.37 s

print(f"Mean Jolly-Hampton CV: {CV_JH_mean:.4f}")

Mean Jolly-Hampton CV: 0.1328

Obtain Kriging Mesh Data#

Access Kriging mesh object#

Reads mesh data files specified by survey_2019

krig_mesh = survey_2019.get_kriging_mesh()

Plot the transects and smoothed isobath contour, and optionally the mesh points#

plot_layered_points generates an interactive map using the Folium package
Transect points are represented by a changing color gradient
Smoothed contour points (200m isobath) are in blue
Here we choose not to plot the mesh points to prevent this notebook from getting too big. To plot the mesh points, omit the plot_mesh_points argument or use plot_mesh_points=True. Mesh points will be in gray

fmap = plot_layered_points(krig_mesh, plot_mesh_points=False)

fmap

Make this Notebook Trusted to load map: File -> Trust Notebook

Apply coordinate transformations#

Longitude transformation
Lat/Lon to distance

Transect points#

krig_mesh.apply_coordinate_transformation(coord_type='transect')

Mesh points#

krig_mesh.apply_coordinate_transformation(coord_type='mesh')

# plot the transformed mesh points 
plt.plot(krig_mesh.transformed_mesh_df.x_mesh, 
         krig_mesh.transformed_mesh_df.y_mesh, 'r*', markersize=1.25)
plt.show()

Compute biomass density Semi-Variogram and fit a model#

Compute the normalized semi-variogram using the areal biomass density
Fit a model to the semi-variogram values

Compute the semi-variogram#

Initialize semi-variogram calculation#

Transformed transect points
Parameters specific to semi-variogram algorithm

semi_vario = survey_2019.get_semi_variogram(
    krig_mesh,
    params=dict(nlag=30, lag_res=0.002)
)

Compute the normalized semi-variogram#

%%time
semi_vario.calculate_semi_variogram()

CPU times: user 3.8 s, sys: 3.03 s, total: 6.83 s
Wall time: 4.43 s

Fit a model to the semi-variogram#

A widget to easily fit the model

semi_vario.get_widget()

Perform Ordinary Kriging of biomass density#

transformed mesh points
semi-variogram model
areal biomass density

Initialize Kriging routine#

kriging_params = dict(
    # kriging parameters
    k_max=10,
    k_min=3,
    R=0.0226287,
    ratio=0.001,
    
    # parameters for semi-variogram model
    s_v_params={'nugget': 0.0, 'sill': 0.95279, 'ls': 0.0075429,
                'exp_pow': 1.5, 'ls_hole_eff': 0.0},
    
    # grab appropriate semi-variogram model
    s_v_model=SV.generalized_exp_bessel
)

# uncomment to use widget values 
# kriging_params.update(semi_vario.get_params_for_kriging())

# initalize kriging routine
krig = survey_2019.get_kriging(kriging_params)

Perform Kriging#

Generates total biomass at mesh points
Computes the total biomass estimate of Kriging output in kmt, where kmt is \(10^3\) metric tons, which is equivalent to \(10^6\) kg

%%time
krig.run_biomass_kriging(krig_mesh)

CPU times: user 5.07 s, sys: 6.15 s, total: 11.2 s
Wall time: 6.42 s

# compute variables at the Kriging mesh points
krig.compute_kriging_variables()

Compute abundance and biomass for each length and age bin, over mesh points#

survey_2019.compute_length_age_variables(data="kriging")

survey_2019.bio_calc.kriging_bin_abundance_male_df.head()

	age_bin_1	age_bin_2	age_bin_3	age_bin_4	age_bin_5	age_bin_6	age_bin_7	age_bin_8	age_bin_9	age_bin_10	...	age_bin_14	age_bin_15	age_bin_16	age_bin_17	age_bin_18	age_bin_19	age_bin_20	age_bin_21	age_bin_22	un-aged
len_bin_2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_6	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_8	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_10	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

5 rows × 23 columns

survey_2019.bio_calc.kriging_bin_biomass_df.head()

	age_bin_1	age_bin_2	age_bin_3	age_bin_4	age_bin_5	age_bin_6	age_bin_7	age_bin_8	age_bin_9	age_bin_10	...	age_bin_14	age_bin_15	age_bin_16	age_bin_17	age_bin_18	age_bin_19	age_bin_20	age_bin_21	age_bin_22	un-aged
len_bin_2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_6	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_8	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
len_bin_10	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

5 rows × 23 columns

Convert biomass estimate to kmt and display results#

# create variables to improve readability
krig_results = survey_2019.bio_calc.kriging_results_gdf
krig_results_male = survey_2019.bio_calc.kriging_results_male_gdf
krig_results_female = survey_2019.bio_calc.kriging_results_female_gdf

# convert to kmt
krig_results.biomass_adult = 1e-6 * krig_results.biomass_adult
krig_results_male.biomass_adult = 1e-6 * krig_results_male.biomass_adult
krig_results_female.biomass_adult = 1e-6 * krig_results_female.biomass_adult

# compute and display total biomass estimates 
print("Kriged Adult Biomass Estimates:")
print(f"Total: {krig_results.biomass_adult.sum():.3f} kmt")
print(f"Total Male: {krig_results_male.biomass_adult.sum():.3f} kmt")
print(f"Total Female: {krig_results_female.biomass_adult.sum():.3f} kmt")

Kriged Adult Biomass Estimates:
Total: 1725.033 kmt
Total Male: 834.250 kmt
Total Female: 890.783 kmt

Plot Kriged Biomass estimate in kmt#

Here we use the argument greater_than_0=True to limit the mesh biomass points that are plotted to ones with biomass value > 0, to prevent this notebook from getting too big. To plot all mesh points, omit the greater_than_0 argument or use greater_than_0=False.

plot_kriging_results(krig_results, krig_field_name="biomass_adult", greater_than_0=True)

Make this Notebook Trusted to load map: File -> Trust Notebook

EchoPro

EchoPro Python Workflow

Contents

EchoPro Python Workflow#

Import libraries and configure the Jupyter notebook#

Set up EchoPro for a specific survey year#

Initialize EchoPro object using configuration files#

Load and process input data#

Compute transect biomass and abundance, and corresponding densities#

Compute abundance and biomass for each length and age bin, over transects#

Jolly-Hampton CV Analysis#

Obtain Kriging Mesh Data#

Access Kriging mesh object#

Plot the transects and smoothed isobath contour, and optionally the mesh points#

Apply coordinate transformations#

Transect points#

Mesh points#

Compute biomass density Semi-Variogram and fit a model#

Compute the semi-variogram#

Initialize semi-variogram calculation#

Compute the normalized semi-variogram#

Fit a model to the semi-variogram#

Perform Ordinary Kriging of biomass density#

Initialize Kriging routine#

Perform Kriging#

Compute abundance and biomass for each length and age bin, over mesh points#

Convert biomass estimate to kmt and display results#

Plot Kriged Biomass estimate in kmt#